Fixes #26670: Resolve QuickSight column aliases to correct upstream column in lineage by mohitjeswani01 · Pull Request #27301 · open-metadata/OpenMetadata

mohitjeswani01 · 2026-04-12T21:54:46Z

Description

I worked on this because QuickSight datasets using CustomSql with column
aliases (e.g. SELECT id AS relation_id) were producing incorrect
column-level lineage. OpenMetadata was matching by column name instead
of tracing the alias back through the SQL expression — so relation_id
would get linked to any other upstream column named relation_id in the
catalog, rather than the actual source column id.

The fix was simpler than it looks — LineageParser was already being
instantiated with the CustomSql query in _yield_lineage_from_query(),
but its column_lineage property was never used for column-level lineage.
The code was falling back to name-based matching instead.

What I did:

Added _build_column_lineage_from_parser() method that uses
lineage_parser.column_lineage to resolve alias mappings correctly
Used src_col._parent to filter by parent table — prevents wrong
lineage when multiple upstream tables share the same column name
Extracted column name from raw_name via .split(".")[-1] to handle
fully-qualified column names from the parser
Guarded against single-element tuple edge cases (len(col_pair) < 2)
Added graceful fallback to name-based matching when parser returns
no column lineage (SQL too complex or parsing failed)
Added get_column_fqn import which was only in the base class before

Type of change:

Bug fix

Checklist:

I have read the CONTRIBUTING document.
My PR title is Fixes #26670: Resolve QuickSight column aliases to correct upstream column in lineage
I have commented on my code, particularly in hard-to-understand areas.
For JSON Schema changes: I updated the migration scripts or explained why it is not needed.
I have added a test that covers the exact scenario we are fixing.

Tests added (test_quicksight.py):

test_build_column_lineage_from_parser_resolves_alias — verifies alias resolution
test_build_column_lineage_from_parser_multi_table_filters_correctly — verifies parent table filtering prevents wrong lineage in multi-table joins
test_build_column_lineage_from_parser_falls_back_when_empty — verifies graceful fallback to name-based matching

github-actions · 2026-04-12T21:55:15Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Copilot

Pull request overview

Resolves incorrect QuickSight CustomSql column-level lineage when projected columns use aliases (e.g., SELECT id AS relation_id) by leveraging LineageParser.column_lineage mappings instead of falling back to name-based matching.

Changes:

Added _build_column_lineage_from_parser() to build ColumnLineage entries from SQL parser column mappings with parent-table filtering and fallback behavior.
Updated _yield_lineage_from_query() to use the new parser-based column lineage builder.
Added unit tests covering alias resolution, multi-table filtering, and fallback behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
`ingestion/src/metadata/ingestion/source/dashboard/quicksight/metadata.py`	Builds column-level lineage using SQL parser alias mappings and uses it during lineage emission.
`ingestion/tests/unit/topology/dashboard/test_quicksight.py`	Adds unit tests to validate alias resolution, multi-table filtering, and fallback behavior.

github-actions · 2026-04-12T22:06:10Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

mohitjeswani01 · 2026-04-12T22:09:22Z

All bot review comments from Copilot and Gitar Bot have been addressed in commit 106c664:

_parent iterable handling — getattr + list normalisation + any() check
Added test_build_column_lineage_from_parser_iterable_parent test (order 12)
Added logger.debug() when _parent is None for visibility
Fixed Python checkstyle violation — f-string line was 97 chars, now split under 88

@harshach @pmbrull could you please add the safe-to-test label? Thank you! 🙏

github-actions · 2026-04-13T12:10:23Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Copilot · 2026-04-13T12:15:38Z

+    @pytest.mark.order(12)
+    def test_build_column_lineage_from_parser_iterable_parent(self):
+        """
+        When src_col._parent is an iterable of parent tables (as some
+        parser outputs produce), _build_column_lineage_from_parser must


The new tests use @pytest.mark.order(12) and then later @pytest.mark.order(11), which makes the ordered test sequence harder to follow and can be confusing when diagnosing failures. Consider keeping the order markers monotonically increasing in the file (swap these two order values, or reorder the tests to match).

Done. Corrected the @pytest.mark.order tags to maintain a monotonically increasing sequence.

PubChimps · 2026-04-16T01:38:26Z

@mohitjeswani01 could you take a look at copilots last 2 comments?

mohitjeswani01 · 2026-04-16T03:02:29Z

@mohitjeswani01 could you take a look at copilots last 2 comments?

yes @PubChimps i will definitely fix those and also check the CI failures!

github-actions · 2026-04-16T20:45:43Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

github-actions · 2026-04-16T20:46:33Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

mohitjeswani01 · 2026-04-16T20:49:36Z

@PubChimps I have addressed the Copilot bot findings. The fallback logic in metadata.py has been updated to prevent manufacturing incorrect cross-table lineage, and the corresponding regression test is added. The pytest order markers in test_quicksight.py are also corrected to a monotonic sequence.

The branch is rebased and ready. Could you please add the safe to test label so I can verify the CI pipelines? Thanks.🙏

… lineage Fixes open-metadata#26670

… also fix bot comments

…orrect pytest ordering

…iedEntityName wrapping

github-actions · 2026-04-29T17:53:11Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

…est mocks

github-actions · 2026-04-29T18:26:11Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

gitar-bot · 2026-04-29T18:27:13Z

Code Review ✅ Approved 2 resolved / 2 findings

QuickSight column aliases are now correctly resolved to their upstream lineage, addressing issues with parent-table filters and incomplete test coverage. No further issues found.

✅ 2 resolved

✅ Edge Case: No parent-table filter when _parent is None in multi-table SQL

📄 ingestion/src/metadata/ingestion/source/dashboard/quicksight/metadata.py:300-307
When src_col._parent is None (line 300), the parent-table filter is skipped entirely. For single-table queries this is correct, but if the parser fails to populate _parent for a multi-table JOIN query, the same column pair will pass the filter for every from_entity in the outer loop of _yield_lineage_from_query, potentially producing duplicate or incorrect column lineage.

get_column_fqn provides a secondary guard (the column must exist in the entity), so impact is limited to cases where both upstream tables have a column with the same name. This is unlikely to cause real issues given the fallback, but worth being aware of.

✅ Bug: Test doesn't exercise parent-table name comparison it claims to

📄 ingestion/tests/unit/topology/dashboard/test_quicksight.py:489-503
In test_build_column_lineage_no_fallback_when_parser_has_global_lineage, other_src_col._parent is assigned a bare MagicMock() (line 492). In the production code (metadata.py line 280), hasattr(parent, "__iter__") is True for MagicMock, so list(parent) is called — which yields an empty list. The any(...) filter at line 284 then returns False and continue is hit, meaning the pair is skipped due to an empty parent list, not because "users_table" != "orders_table" as the test docstring implies. The __str__ configured at line 493 is never invoked.

The test still validates the key invariant (no fallback when parser has global lineage), and test_build_column_lineage_from_parser_multi_table_filters_correctly does properly test the name comparison using _FakeTable. So there's no gap in coverage, but this test is misleading about what it exercises.

Options

Display: compact → Showing less information.

Comment with these commands to change:

`Compact`
`gitar display:verbose`

_{Was this helpful? React with 👍 / 👎 | Gitar}

mohitjeswani01 · 2026-04-29T18:28:49Z

@PubChimps , @harshach sir i have solved the merge conflicts also resolved bots comments and ran the tests locally could you please add a safe to test label and review this ? thanks 🙏!

mohitjeswani01 requested a review from a team as a code owner April 12, 2026 21:54

Copilot AI review requested due to automatic review settings April 12, 2026 21:54

Copilot started reviewing on behalf of mohitjeswani01 April 12, 2026 21:55 View session

Copilot AI reviewed Apr 12, 2026

View reviewed changes

Comment thread ingestion/src/metadata/ingestion/source/dashboard/quicksight/metadata.py Outdated

Comment thread ingestion/tests/unit/topology/dashboard/test_quicksight.py

gitar-bot Bot reviewed Apr 12, 2026

View reviewed changes

Comment thread ingestion/src/metadata/ingestion/source/dashboard/quicksight/metadata.py Outdated

Copilot AI review requested due to automatic review settings April 13, 2026 12:09

Copilot started reviewing on behalf of mohitjeswani01 April 13, 2026 12:10 View session

Copilot AI reviewed Apr 13, 2026

View reviewed changes

This was referenced Apr 16, 2026

Fix/issue 26670 quicksight column aliases #27047

Closed

fix(quicksight): resolve column aliases to correct upstream column in lineage (#26670) #27031

Closed

mohitjeswani01 force-pushed the fix/26670-quicksight-column-alias-lineage branch from b501e67 to edfdf80 Compare April 16, 2026 20:45

mohitjeswani01 added 4 commits April 29, 2026 22:41

fix(quicksight): resolve column aliases to correct upstream column in…

1c5f127

… lineage Fixes open-metadata#26670

fix(quicksight): handle iterable _parent and fix checkstyle violation…

37739a4

… also fix bot comments

Address Copilot review: Fix multi-table column lineage fallback and c…

6d190f0

…orrect pytest ordering

fix: resolve rebase conflicts and fix test assertions for FullyQualif…

9ae5cbc

…iedEntityName wrapping

mohitjeswani01 force-pushed the fix/26670-quicksight-column-alias-lineage branch from 7735d55 to 9ae5cbc Compare April 29, 2026 17:52

Copilot AI review requested due to automatic review settings April 29, 2026 17:52

Copilot started reviewing on behalf of mohitjeswani01 April 29, 2026 17:53 View session

gitar-bot Bot reviewed Apr 29, 2026

View reviewed changes

Comment thread ingestion/tests/unit/topology/dashboard/test_quicksight.py

Copilot AI reviewed Apr 29, 2026

View reviewed changes

Comment thread ingestion/src/metadata/ingestion/source/dashboard/quicksight/metadata.py Outdated

fix(quicksight): address bot reviews for empty parent iterables and t…

f5cd2cb

…est mocks

Conversation

mohitjeswani01 commented Apr 12, 2026

Description

Type of change:

Checklist:

Uh oh!

github-actions Bot commented Apr 12, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 12, 2026

Uh oh!

mohitjeswani01 commented Apr 12, 2026

Uh oh!

github-actions Bot commented Apr 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

mohitjeswani01 Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

PubChimps commented Apr 16, 2026

Uh oh!

mohitjeswani01 commented Apr 16, 2026

Uh oh!

github-actions Bot commented Apr 16, 2026

Uh oh!

github-actions Bot commented Apr 16, 2026

Uh oh!

mohitjeswani01 commented Apr 16, 2026

Uh oh!

github-actions Bot commented Apr 29, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

github-actions Bot commented Apr 29, 2026

Uh oh!

gitar-bot Bot commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mohitjeswani01 commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gitar-bot Bot commented Apr 29, 2026 •

edited

Loading

mohitjeswani01 commented Apr 29, 2026 •

edited

Loading